Statistical Model Selection Criteria Research Articles

Combination of a global-search method with model selection criteria for the ellipsometric data evaluation of DLC coatings

Abstract A method for the evaluation of experimental data from spectroscopic ellipsometry is proposed which combines the global-search optimization algorithm with statistical model selection criteria. The hybrid genetic-gradient search algorithm (HGGA) is applied to find the optical parameters and thickness of a diamond-like carbon (DLC) coating deposited on SW7M stainless steel. Akaike and Bayesian information criteria are used to evaluate the different dielectric function models. The method is able to find optical model parameters even in case of a limited initial knowledge about the material optical constants. At the same time, the optimal dielectric function model for the description of the material optical properties can be selected unambiguously from the set of candidate models.

Latent profile analysis of human values: What is the optimal number of clusters?

Latent Profile Analysis (LPA) is a method to extract homogeneous clusters characterized by a common response profile. Previous works employing LPA to human value segmentation tend to select a small number of moderately homogeneous clusters based on model selection criteria such as Akaike information criterion, Bayesian information criterion and Entropy. The question is whether a small number of clusters is all that can be gleaned from the data. While some studies have carefully compared different statistical model selection criteria, there is currently no established criteria to assess if an increased number of clusters generates meaningful theoretical insights. This article examines the content and meaningfulness of the clusters extracted using two algorithms: Variational Bayesian LPA and Maximum Likelihood LPA. For both methods, our results point towards eight as the optimal number of clusters for characterizing distinctive Schwartz value typologies that generate meaningful insights and predict several external variables.

Quantifying and Modeling the Cumulative Impact of Changes Order for the Projects in Iraq

This research investigates the change orders projects of hospital construction in Iraq, for several reasons. Because these types of projects have a relatively long duration, which leads to a change in some requirements or the emergence of new developments with the owner or gets a new technical development in this field, also these projects contain a large number of documents, specifications, plans, bill of quantities, and a large number of engineers with various specializations participate in preparing them, which makes the probability of change possible due to the large work and poor coordination between those specializations or the development of new government regulations. This current research is based on a database of eleven projects for hospitals. The researcher used combined methodology: the qualitative method represented by interviews of many construction professionals, and the quantitative method represented by questionnaire surveys and gathering data through a survey the documents of change orders for the projects. In this study, we adopted the analytical statistical approach by used F-tests and then used a one - way ANOVA analysis. The researcher managed to reach mathematical models that can predict the added cost of change requests that occur through the planning or implementation phase for the project, to avoid or minimize project changes. A series of statistical model selection criteria were applied to carefully identify the best models, for ease of use of search results and to obtain a tool that can be used by the parties to the project.

A generative learning model for saccade adaptation.

Plasticity in the oculomotor system ensures that saccadic eye movements reliably meet their visual goals—to bring regions of interest into foveal, high-acuity vision. Here, we present a comprehensive description of sensorimotor learning in saccades. We induced continuous adaptation of saccade amplitudes using a double-step paradigm, in which participants saccade to a peripheral target stimulus, which then undergoes a surreptitious, intra-saccadic shift (ISS) as the eyes are in flight. In our experiments, the ISS followed a systematic variation, increasing or decreasing from one saccade to the next as a sinusoidal function of the trial number. Over a large range of frequencies, we confirm that adaptation gain shows (1) a periodic response, reflecting the frequency of the ISS with a delay of a number of trials, and (2) a simultaneous drift towards lower saccade gains. We then show that state-space-based linear time-invariant systems (LTIS) represent suitable generative models for this evolution of saccade gain over time. This state-equation algorithm computes the prediction of an internal (or hidden state-) variable by learning from recent feedback errors, and it can be compared to experimentally observed adaptation gain. The algorithm also includes a forgetting rate that quantifies per-trial leaks in the adaptation gain, as well as a systematic, non-error-based bias. Finally, we study how the parameters of the generative models depend on features of the ISS. Driven by a sinusoidal disturbance, the state-equation admits an exact analytical solution that expresses the parameters of the phenomenological description as functions of those of the generative model. Together with statistical model selection criteria, we use these correspondences to characterize and refine the structure of compatible state-equation models. We discuss the relation of these findings to established results and suggest that they may guide further design of experimental research across domains of sensorimotor adaptation.

Validation of the statistical parameters and model selection criteria of the benchmark dose methods for the evaluation of various endpoints in repeated-dose toxicity studies

Validation of the statistical parameters and model selection criteria of the benchmark dose methods for the evaluation of various endpoints in repeated-dose toxicity studies

The importance of distribution-choice in modeling substance use data: a comparison of negative binomial, beta binomial, and zero-inflated distributions

Background: It is important to correctly understand the associations among addiction to multiple drugs and between co-occurring substance use and psychiatric disorders. Substance-specific outcomes (e.g. number of days used cannabis) have distributional characteristics which range widely depending on the substance and the sample being evaluated. Objectives: We recommend a four-part strategy for determining the appropriate distribution for modeling substance use data. We demonstrate this strategy by comparing the model fit and resulting inferences from applying four different distributions to model use of substances that range greatly in the prevalence and frequency of their use. Methods: Using Timeline Followback (TLFB) data from a previously-published study, we used negative binomial, beta-binomial and their zero-inflated counterparts to model proportion of days during treatment of cannabis, cigarettes, alcohol, and opioid use. The fit for each distribution was evaluated with statistical model selection criteria, visual plots and a comparison of the resulting inferences. Results: We demonstrate the feasibility and utility of modeling each substance individually and show that no single distribution provides the best fit for all substances. Inferences regarding use of each substance and associations with important clinical variables were not consistent across models and differed by substance. Conclusion: Thus, the distribution chosen for modeling substance use must be carefully selected and evaluated because it may impact the resulting conclusions. Furthermore, the common procedure of aggregating use across different substances may not be ideal.

Paradigm free mapping with sparse regression automatically detects single-trial functional magnetic resonance imaging blood oxygenation level dependent responses.

The ability to detect single trial responses in functional magnetic resonance imaging (fMRI) studies is essential, particularly if investigating learning or adaptation processes or unpredictable events. We recently introduced paradigm free mapping (PFM), an analysis method that detects single trial blood oxygenation level dependent (BOLD) responses without specifying prior information on the timing of the events. PFM is based on the deconvolution of the fMRI signal using a linear hemodynamic convolution model. Our previous PFM method (Caballero-Gaudes et al., 2011: Hum Brain Mapp) used the ridge regression estimator for signal deconvolution and required a baseline signal period for statistical inference. In this work, we investigate the application of sparse regression techniques in PFM. In particular, a novel PFM approach is developed using the Dantzig selector estimator, solved via an efficient homotopy procedure, along with statistical model selection criteria. Simulation results demonstrated that, using the Bayesian information criterion to select the regularization parameter, this method obtains high detection rates of the BOLD responses, comparable with a model-based analysis, but requiring no information on the timing of the events and being robust against hemodynamic response function variability. The practical operation of this sparse PFM method was assessed with single-trial fMRI data acquired at 7T, where it automatically detected all task-related events, and was an improvement on our previous PFM method, as it does not require the definition of a baseline state and amplitude thresholding and does not compromise on specificity and sensitivity.

A Review of Statistical Model Selection Criteria: Application to Prediction in Regression, Histograms, and Finite Mixture Models

Some model-selection criteria for choosing among a set of alternative models are reviewed. Particular model-selection problems considered here include choice of a regression equation for prediction, the number of bins for a histogram, and the number of component p.d.f.s in a finite mixture model.Several general methods of scoring the choices in such problems are considered. Minimum description length and penalized likelihood criteria are discussed, in particular AIC (Akaike's Information Criterion), BIC (Bayesian Information Criterion) and KIC (Kashyap's Information Criterion). Interpretation of BIC and KIC in terms of posterior probabilities of alternative models is given. Averaging a prediction or classification over models is considered.

Using statistical model selection criteria to discriminate nonsubjectively between hypotheses about physiologic mechanisms underlying experimental observations: a practical example

Using statistical model selection criteria to discriminate nonsubjectively between hypotheses about physiologic mechanisms underlying experimental observations: a practical example

Tactical Industry Allocation and Model Uncertainty

AbstractWe use Bayesian model averaging to analyze industry return predictability in the presence of model uncertainty. The posterior analysis shows the importance of inflation and earnings yield in predicting industry returns. The out‐of‐sample performance of the Bayesian approach is, in general, superior to that of other statistical model selection criteria. However, the out‐of‐sample forecasting power of a naive i.i.d. forecast is similar to the Bayesian forecast. A variance decomposition into model risk, estimation risk, and forecast error shows that model risk is less important than estimation risk.

A comparative study of model selection criteria for the number of signals

The performance of six existing model selection criteria is compared, which are commonly used in time series and regression analysis, when they are applied to the problem of the number of signals in the multiple signal classification (MUSIC) method. The five criteria are Akaike Information Criterion (AIC), Hannan and Quinn Criterion, Bayesian Information Criterion (BIC), Corrected AIC (AICc) and the recently introduced Vector Corrected Kullback Information Criterion (KICvc) and Weighted-Average Information Criterion (WIC). The general form of the above information criteria consists of a log likelihood function expressed in terms of the eigenvalues of the sample covariance matrix and a unique penalty term. In our estimation procedure, the number of signals is obtained by minimising each of the above criteria. Several simulated data sets, including a linear antenna array data set, are adopted for the comparison purpose. The authors show that, in simple MUSIC additive white noise model, for small sample size n, WIC performs nearly as well as AICc and outperforms other criteria, and for moderately large to large n, WIC performs nearly as well as BIC and outperforms other criteria. Therefore when the authors are not certain of the relative sample size, WIC may be a practical alternative to any criterion. The main purpose is to draw the attention and interests of signal processing researchers to adopt more recent statistical model selection criteria, such as WIC, in general signal processing problems.

The Commitment to Development Index: An Information Theory approach

Using statistical model selection criteria rooted in Information Theory which penalise complexity we show that there is little justification for relaxing the equal weights assumption underlying the Center for Global Development's, Commitment to Development Index (CDI). The CDI is a composite index which combines metrics of aid, trade, investment, migration, environment, security and technology. A survey of researchers recently concluded that the CDI should not weight each of these six components equally. Specifically, it suggested that: trade and investment should be weighted higher; migration and aid should be weighted lower; with peacekeeping and environment not statistically different from equal weights. Generating hypothetical data around the weights proposed by the results of this survey we test an equally weighted CDI against two unequally weighted alternatives. Although the unequally weighted alternatives provide a superior goodness-of-fit to these hypothetical datasets, this is more than counteracted by the increased model complexity associated with these unequally weighted models according to most model selection criteria. The results of our analysis suggest that the CDI should not diverge from its equal weights assumption.

Tactical Industry Allocation and Model Uncertainty

We use Bayesian model averaging to analyze the sample evidence on industry return predictability within the U.S. stock market in the presence of model uncertainty. The posterior analysis shows the importance of inflation and earnings yield in predicting industry returns. The out-of-sample performance of the Bayesian approach is, in general, superior to that of other statistical model selection criteria. However, the out-of-sample forecasting power of a naive iid forecast is similar to the Bayesian forecast. A variance decomposition into model risk, estimation risk, and forecast error shows that model risk is less important than estimation risk.

An automated process for building reliable and optimal in vitro/in vivo correlation models based on Monte Carlo simulations

Many mathematical models have been proposed for establishing an in vitro/in vivo correlation (IVIVC). The traditional IVIVC model building process consists of 5 steps: deconvolution, model fitting, convolution, prediction error evaluation, and cross-validation. This is a time-consuming process and typically a few models at most are tested for any given data set. The objectives of this work were to (1) propose a statistical tool to screen models for further development of an IVIVC, (2) evaluate the performance of each model under different circumstances, and (3) investigate the effectiveness of common statistical model selection criteria for choosing IVIVC models. A computer program was developed to explore which model(s) would be most likely to work well with a random variation from the original formulation. The process used Monte Carlo simulation techniques to build IVIVC models. Data-based model selection criteria (Akaike Information Criteria [AIC], R2) and the probability of passing the Food and Drug Administration "prediction error" requirement was calculated. To illustrate this approach, several real data sets representing a broad range of release profiles are used to illustrate the process and to demonstrate the advantages of this automated process over the traditional approach. The Hixson-Crowell and Weibull models were often preferred over the linear. When evaluating whether a Level A IVIVC model was possible, the model selection criteria AIC generally selected the best model. We believe that the approach we proposed may be a rapid tool to determine which IVIVC model (if any) is the most applicable.

Market Timing And Model Uncertainty: An Exploratory Study For The Swiss Stock Market

We use statistical model selection criteria and Avramov's (2002) Bayesian model averaging approach to analyze the sample evidence of stock market predictability in the presence of model uncertainty. The empirical analysis for the Swiss stock market is based on a number of predictive variables found important in previous studies of return predictability. We find that it is difficult to discard any predictive variable as completely worthless, but that the posterior probabilities of the individual forecasting models as well as the cumulative posterior probabilities of the predictive variables are time-varying. Moreover, the estimates of the posterior probabilities are not robust to whether the predictive variables are stochastically detrended or not. The decomposition of the variance of predicted future returns into the components parameter uncertainty, model uncertainty, and the uncertainty attributed to forecast errors indicates that the respective contributions strongly depend on the time period under consideration and the initial values of the predictive variables. In contrast to AVRAMOV (2002), model uncertainty is generally not more important than parameter uncertainty. Finally, we demonstrate the implications of model uncertainty for market timing strategies. In general, our results do not indicate any reliable out-of-sample return predictability. Among the predictive variables, the dividend-price ratio exhibits the worst external validation on average. Again in contrast to AVRAMOV (2002), our analysis suggests that the out-of-sample performance of the Bayesian model averaging approach is not superior to the statistical model selection criteria. Consequently, model averaging does not seem to help improve the performance of the resulting short-term market timing strategies.

Statistical Model Selection Criteria and the Philosophical Problem of Underdetermination

I discuss the philosophical significance of the statistical model selection criteria, in particular their relevance for philosophical problems of underdetermination. I present an easily comprehensible account of their simplest possible application and contrast it with their application to curve‐fitting problems. I embed philosophers' earlier discussion concerning the situations in which the criteria yield implausible results into a more general framework. Among other things, I discuss a difficulty which is related to the so‐called subfamily problem, and I show that it has analogies in all legitimate applications of the model selection criteria, and that an analogy of Goodman's new riddle of induction can be formulated in only some of their applications.

A Semiparametric Deconvolution Model to Establish In Vivo–In Vitro Correlation Applied to OROS Oxybutynin

In vitro–in vivo correlation (IVIVC) models may be used to predict in vivo drug concentration–time profiles given in vitro release characteristics of a drug. This prediction is accomplished by incorporating in vitro release characteristics as an input function (Avitro) to a pharmacokinetics model. This simple approach often results in biased predictions of observed in vivo drug concentrations, and it can result in rejecting IVIVC. To solve this problem we propose a population IVIVC model that incorporates the in vitro information and allows one to quantify possibly changed in vivo release characteristic. The model assumes linear kinetics and describes the in vivo release as a sum of Avitro and a nonparametric function (Ad, a spline) representing the difference in release due to in vivo conditions. The function Avitro and its variability enter the model as a prior distribution. The function Ad is estimated together with its intersubject variability. The number of parameters associated with Ad defines the model: no parameters indicates perfect IVIVC, a large number of parameters indicates poor IVIVC. The number of parameters is determined using statistical model selection criteria. We demonstrate the approach to solve the IVIVC problem of an oral extended release oxybutynin form (OROS), administered in three pharmacokinetic studies. These studies present a particular challenging case; that is, the relative bioavailability for the OROS administration is >100% compared with that of the immediate‐release form. The result of our modeling shows that the apparent lack of IVIVC can be overcome: in vivo concentration can be predicted (within or across data sets) based on in vitro release rate together with a simple form of systematic deviation from the in vitro release. © 2001 Wiley‐Liss, Inc. and the American Pharmaceutical Association J Pharm Sci 90:702–712, 2001

Statistical Model Selection Criteria and Bayesianism

Two Bayesian approaches to choosing between statistical models are contrasted. One of these is an approach which Bayesian statisticians regularly use for motivating the use of AIC, BIC, and other similar model selection criteria, and the other one is a new approach which has recently been proposed by Bandyopadhayay, Boik, and Basu. The latter approach is criticized, and the basic ideas of the former approach are presented in a way that makes them accessible to a philosophical audience. It is also pointed out that the former approach establishes a new, philosophically interesting connection between the notions of simplicity and informativeness.

LINEAR MIXED-EFFECT MULTIVARIATE ADAPTIVE REGRESSION SPLINES APPLIED TO NONLINEAR PHARMACOKINETICS DATA

In a frequently performed pharmacokinetics study, different subjects are given different doses of a drug. After each dose is given, drug concentrations are observed according to the same sampling design. The goal of the experiment is to obtain a representation for the pharmacokinetics of the drug, and to determine if drug concentrations observed at different times after a dose are linear in respect to dose. The goal of this paper it to obtain a representation for concentration as a function of time and dose, which (a) makes no assumptions on the underlying pharmacokinetics of the drug; (b) takes into account the repeated measure structure of the data; and (c) detects nonlinearities in respect to dose. To address (a) we use a multivariate adaptive regression splines representation (MARS), which we recast into a linear mixed-effects model, addressing (b). To detect nonlinearity we describe a general algorithm that obtains nested (mixed-effect) MARS representations. In the pharmacokinetics application, the algorithm obtains representations containing time, and time and dose, respectively, with the property that the bases functions of the first representation are a subset of the second. Standard statistical model selection criteria are used to select representations linear or nonlinear in respect to dose. The method can be applied to a variety of pharmacokinetics (and pharmacodynamic) preclinical and phase I–III trials. Examples of applications of the methodology to real and simulated data are reported. Work supported by NIH Grant GM 51197 and GM 57193.

Application of statistical model selection criteria to the Stock Synthesis assessment program

Statistical modeling involves building sufficiently complex models to represent the system being investigated. Overly complex models lead to imprecise parameter estimates, increase the subjective role of the modeler, and can distort the perceived characteristics of the system under investigation. One approach for controlling the tendency to increased complexity and subjectivity is to use model selection criteria that account for these factors. The effectiveness of two selection criteria was tested in an application with the stock assessment program known as Stock Synthesis. This program, which is often used on the U.S. west coast to assess the status of exploited marine fish stocks, can handle multiple data sets and mimic highly complex population dynamics. The Akaike information criterion and Schwarz's Bayesian information criterion are criteria that satisfy the fundamental principles of model selection: goodness-of-fit, parsimony, and objectivity. Their ability to select the correct model form and produce accurate estimates was evaluated in Monte Carlo experiments with the Stock Synthesis program. In general, the Akaike information criterion and the Bayesian information criterion had similar performance in selecting the correct model, and they produced comparable levels of accuracy in their estimates of ending stock biomass.